A Scanning Electron Microscope (SEM) as one of charged particle beam devices is a device to scan over a minute object as a sample with a narrow-focused electron beam so as to let the sample emit secondary electrons or the like, and to observe, examine or measure, for example, the minute object based on the secondary electrons or the like. An energy filter of a SEM is made up of a plurality of electrodes, among which electric field is formed to limit the passage of the secondary electrons or the like emitted from the sample, whereby the energy filter is used for selectively detecting electrons having specific energy. Patent Literature 1 describes a Wien filter to separate electrons emitted from a sample away from the optical axis and to introduce the electrons to a detector via an energy filter. Patent Literature 2 discloses a scanning electron microscope provided with a focusing element that focuses the trajectory of electrons emitted from a sample to correct the trajectory, thus achieving efficient detection of secondary electrons. Patent Literature 3 discloses a retarding field electrode arrangement that forms a potential barrier at an objective lens to retard particles emitted from a sample.